Skip to content

Add opt-in rhpkg releases with destination tag reconciliation - #458

Open
Odilhao wants to merge 2 commits into
theforeman:masterfrom
Odilhao:feature/rhpkg-releaser
Open

Add opt-in rhpkg releases with destination tag reconciliation#458
Odilhao wants to merge 2 commits into
theforeman:masterfrom
Odilhao:feature/rhpkg-releaser

Conversation

@Odilhao

@Odilhao Odilhao commented Sep 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Add a Brew/dist-git release workflow behind an explicit build_package_releaser: rhpkg opt-in. Existing Tito inventories, rel-eng/releasers.conf, custom builders, nightly settings, and setup_sources_git continue to work. Direct Koji and COPR workflows retain their existing behavior.

  • Configure dist-git branches, build targets, RPM contexts, and destination tags through rhpkg_targets.
  • Reuse complete NVRs and reconcile missing destination tags. For missing builds, stage spec sources and patches, upload lookaside sources, push dist-git, and submit through rhpkg.
  • Default rhpkg operations to Brew while preserving explicit executable overrides. Provision and configure rhpkg separately; obal setup retains its existing Tito dependencies.
  • Preserve receipts for asynchronous releases and scratch builds. Validate pending inputs before reuse, allow fresh submissions after failed or cancelled scratch tasks, and preserve receipts on query failures.
  • Submit scratch SRPMs directly through the configured Koji/Brew client without release tags or dist-git changes.
  • Document the new configuration and rollout requirements.

Rollout

Rhpkg is enabled only by the explicit selector. Defining targets alone does not change the backend. build_package_use_koji_build: true continues to select direct Koji submission unless a releaser is explicitly configured.

Tito removal, broader direct Koji tagging changes, and the annex --anything lookup are deferred to separate changes. Tito and setup_sources_git removal should follow a real Brew release and a separately planned major-version migration.

Review fixes

Return structured module errors for Koji tag-verification query failures, extract actual SRPM source bytes on RPM 4 and RPM 6, handle Koji's status-2 unknown-target response, recover concurrent builds by querying their state after submission failure, and check saved scratch task states before resuming.

Validation

  • Python suite and flakes: 226 checks passed on Python 3.12 / CentOS Stream 10 as an unprivileged user.
  • Pylint: 10.00/10.
  • Revised Ansible playbooks and includes passed ansible-lint; custom-module import warnings were emitted.
  • A real RPM 6.0.2 SRPM reproduced the original empty-payload extraction and verified the corrected extraction against the source bytes.
  • git diff --check passed.

No live Brew release or remote dist-git submission was performed.

@Odilhao Odilhao changed the title Add rhpkg releases with automatic Koji tagging Replace Tito releases with rhpkg and automatic Koji tagging Sep 30, 2026
@Odilhao
Odilhao force-pushed the feature/rhpkg-releaser branch 3 times, most recently from d746c60 to fc217f5 Compare September 30, 2026 18:30
zjhuntin
zjhuntin requested changes Sep 30, 2026
View reviewed changes

@zjhuntin zjhuntin left a comment •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found some issues, here ya go:

Comment thread obal/data/roles/build_package/tasks/rhpkg_release.yml
Comment thread obal/data/module_utils/rhpkg.py
@Odilhao
Odilhao force-pushed the feature/rhpkg-releaser branch from fc217f5 to f580f84 Compare September 30, 2026 21:51
ogajduse
ogajduse requested changes Oct 12, 2026
View reviewed changes

@ogajduse ogajduse left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the substantial work here. Before going further on code, I'd like to settle scope and rollout, because as it stands this breaks satellite-packaging on merge: build_package_releaser defaults to rhpkg for any bare build_package_build_system: koji inventory, rel-eng/releasers.conf is no longer read, and obal setup tries to install rhpkg, which is not in Fedora/EPEL/CentOS. Jenkins runs obal from unpinned master, so there is no staged rollout.

Proposal: split this into (a) the rhpkg releaser behind an explicit build_package_releaser: rhpkg opt-in, with tito untouched; (b) tag reconciliation for existing builds (#424's scope); (c) tito and setup_sources_git removal as a separate major-version change once (a) has done a real Brew release; (d) the annex --anything change with a second-clone test. That matches the "just the obal release workflow" scope pcreech described on #380.

Verified defects to fix in whichever part they land in:

  • koji_verify_tag.py:69 catches CalledProcessError, but koji() raises KojiCommandError, so a brew failure crashes the module instead of fail_json.
  • tests/fixtures/rhpkg/mock_client.py:69: cpio -i --to-stdout payload.txt extracts nothing when the member is ./payload.txt (rpm 6 here), so the annex-bytes test compares against sha256(''). It fails on Fedora 43 and is a no-op where it passes.
  • rhpkg.py: the "already been built" recovery can't trigger with --skip-nvr-check always set; the "Unknown build target" check is unreachable because real koji list-targets --name exits 2; a cancelled scratch task with unchanged inputs deadlocks resume().
  • docs/source/actions.rst still documents the tito variables; rhpkg_targets and build_package_releaser are undocumented.

Also please reply to and resolve zjhuntin's two threads; the amend fixed both, but the PR reads as unaddressed.

@Odilhao
Make rhpkg releases opt-in and address review feedback
3c4a5ec 
Preserve Tito and legacy source workflows for a staged rollout.
Fix Koji error handling, target diagnostics, concurrent build recovery,
scratch retries, and SRPM payload extraction. Document rhpkg configuration
and restore legacy regression coverage.

Validation: 226 Python and flakes checks passed; pylint 10.00/10;
Ansible lint passed; extraction verified with a real RPM 6 SRPM.
@Odilhao Odilhao changed the title Replace Tito releases with rhpkg and automatic Koji tagging Add opt-in rhpkg releases with destination tag reconciliation Oct 14, 2026
@Odilhao

Odilhao commented Oct 14, 2026

Copy link
Copy Markdown
Member Author

@ogajduse Addressed in 3c4a5ec. I adopted the staged rollout scope:

  • Rhpkg requires explicit build_package_releaser: rhpkg; targets alone do not select it. Bare Koji inventories retain Tito, and build_package_use_koji_build: true retains direct Koji submission.
  • Tito modules, wrappers, extensions, legacy releaser configuration, nightly settings, and setup_sources_git are restored. obal setup keeps its existing Tito dependencies and no longer attempts to install rhpkg from distribution repositories; the docs describe separate provisioning in the Brew environment.
  • The broader direct Koji tagging changes and annex --anything lookup have been removed from this PR. Rhpkg's destination reconciliation remains confined to the explicit opt-in path. Tito/source-builder removal and the broader annex lookup with a second-clone test are deferred to separate work. No live Brew release has been submitted.

The verified defects are fixed:

  • koji_verify_tag catches KojiCommandError and returns a structured module failure.
  • SRPM extraction accepts both payload.txt and ./payload.txt and rejects empty extraction. A real RPM 6.0.2 SRPM reproduced the original failure and verified the recovered bytes against the source.
  • Unknown-target handling recognizes Koji's actual status-2 response and keeps authentication/query errors distinct.
  • Scratch resumption checks saved task states. Failed or cancelled tasks permit a new submission; active tasks still require matching inputs, and query failures preserve receipts.
  • Concurrent-build recovery queries the requested NVR after a failed submission and reuses COMPLETE or BUILDING results. --skip-nvr-check remains because obal's configured target macro context can differ from rhpkg's local NVR context; recovery no longer depends on the suppressed client error message. Tests cover both build states with a generic hub rejection.
  • docs/source/actions.rst now documents the selector, target fields, executable defaults, prerequisites, and retry behavior.

Validation: 226 Python/flake checks passed on Python 3.12 / CentOS Stream 10 as an unprivileged user; pylint is 10.00/10; changed Ansible playbooks and their includes passed ansible-lint (custom-module import warnings were emitted); git diff --check passed.

I also replied to and resolved @zjhuntin's two inline threads, and updated the PR title and description to match the revised scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zjhuntin zjhuntin zjhuntin requested changes
@ogajduse ogajduse ogajduse requested changes

Assignees

No one assigned

Labels

Waiting on contributor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Odilhao @zjhuntin @ogajduse